5

Regarding raddit search functionality

Submitted by ziq admin in meta (edited )

emma was working at implementing Elasticsearch on the site but we've run into a roadblock.

https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html

Elasticsearch needs a lot of ram to run well. We use just 200mb right now of the 1gb we have available. Upgrading would at least quadruple the costs of running the site.

This is the VPS service we use:

https://www.digitalocean.com/pricing/

We're currently on the $10 a month package but could just as easily be using the $5 a month one with current traffic. If we upgraded to the $40 a month package (4gb ram), it might not be enough still - people seem to be saying 32gb is the ideal minimum for Elasticsearch, with 16gb being acceptable. These VPS packages could be expensive enough to make the site unsustainable in the long term - I doubt I can come up with $320 every month.

Another, far less appealing option is to add duckduckgo search to the site, which wouldn't cost anything but will probobly be a bit shit.

Any other ideas?

Comments

You must log in or register to comment.

4

sancocho wrote

I support duckduckgo, maybe at look to implement subforum specific bangs like !rameta or !ranarchism.

3

devtesla wrote

On the fempire I just got used to doing site:thefempire.org in google and always had a good time. Hopefully that works here and with ddg

2

sudo wrote

I poked around in Raddit's source code a little bit. I should mention that I'm not a PHP programmer.

It looks like emma made something called a "SubmissionRepository", based on a third-party library's "EntityRepository". Guessing by the name, I assume this is a container that stores all of the submissions to the website, and it looks like each submission instance stores its own title, body, submitter, etc. that people would want to search for. So, what's stopping us from implementing a search function that searches through the array of posts, and returns a separate array with only the posts that match the search criteria? Again, I only took a cursory look at the code, and emma knows it much better than I do, but I don't see why we'd need to implement a giant third-party search library when a few functions should do.

3

emma wrote

The *Repository classes are collections of methods that make calls to the database. There's no data stored in there, just custom queries for when the findBy/findOneBy methods from the base class are insufficient.

Elasticsearch was chosen because it's a proper, full text search engine that performs linguistic analysis on the data it indexes and can handle misspellings, figure out what's relevant or not, and so on. I want to avoid a situation where we have a search engine, but it's so shitty that everyone just types site:raddit.me in Google to find what they're looking for (phpBB, I'm looking at you).

It also integrates nicely with the Symfony framework--a bit of configuration, a few lines of code and some HTML templates were all I needed to get it up and running.

But alas, it uses too much RAM, and so it's not an option.

Our best shot would probably be to use PostgreSQL's full text search, but it's not nearly as intuitive, and so finding out how to make use of it and get good results is going to take a while longer.

2

sudo wrote

Thanks for explaining. Elasticsearch does sound much nicer now; it's a shame it's such a memory hog.

2

theemacsshibe wrote (edited )

Okay, could we send donations anywhere for some decent VPS work? I have nothing else to do with my magic internet numbers.

1

ziq wrote (edited )

I just made a patreon but I doubt there are enough people interested in a search function to pay towards it.

I have to look into bitcoin and see if digitalocean accepts them.

But is a search engine worth the expense on such a small site?