Filter shows old data

[mhgolij]

I have an index of products where the category of some items changes every few days. When I filter by category, the returned data shows products that have been removed from the selected category. Does Meilisearch have a cache system that needs to be cleared?

[curquiza]

Hello @mhgolij

No meilisearch does not have any cache system

However, Meilisearch does not process indexing tasks synchronously. Every task is enqueued in a task queue, so are not processed immediately. That's maybe why you don't get the expected result
https://www.meilisearch.com/docs/learn/async/asynchronous_operations

[mhgolij]

Yes! I sure for 100%

[irevoire]

Hey @mhgolij, sorry but I can’t really help here.

I tried to reproduce your issue on the latest version (v1.7) and on the v1.6.2, and it didn’t reproduce.

Here’s what I did; let me know if I did something wrong:

1. Add the document

{
    "id": 1,
    "name": "book 1",
    "categories": {
      "115": "poetry",
      "112": "novel"
    }           
  }

2. Make the categories filterable by sending: { "filterableAttributes": ["categories"] }
3. Run a search request with { "filter": "categories.112 EXISTS" } => the document is returned
4. Update the document by removing the novel field:

{
    "id": 1,
    "name": "book 1",
    "categories": {
      "115": "poetry" 
    }
  }

5. Run the same search request as previously: { "filter": "categories.112 EXISTS" } => the document is NOT returned

If you don’t have more info or a better reproducer, could you try to run my tests and let me know if it works on your side?
If it doesn’t, then maybe you have another issue.

[mhgolij]

Hi, thanks for following up on the problem. One point you may not have noticed is that the filter works correctly on objects with a small number. However, when the number exceeds 55,000, we encounter this problem.

What's interesting is that if we alter the search settings once (for example, by placing the categories in searchableAttributes) to prevent the book from appearing in the results, and then revert the settings to their original state, our filter results appear correctly!

I have no idea why this is happening.

[mhgolij]

As you can see in the pictures, my object has only one category with key 6, and my query filtered objects by categories.108 EXISTS but object appear in result

[irevoire]

Hey, this looks like something else, then.
Are you 100% sure that you have no failed tasks in your task queue?
If you run this command from the task queue documentation, for example,

curl \
  -X GET 'http://localhost:7700/tasks?statuses=failed'

To retrieve all your failed tasks, you should see some stuff falling.

---

Another thing I’m very interested in would be the stats of your index.
Could you do a call to the stats route:

curl \
  -X GET 'http://localhost:7700/stats'

And share its output here; I’m interested in seeing the field distributions of your indexes.

However, when the number exceeds 55,000, we encounter this problem.

My theory is that you’re reaching this limitation of the engine.
You’re trying to store more than 65,536 attributes over all your documents which throws an error because meilisearch stores the field IDs on 16 bits.

Unfortunately, there is no workaround in meilisearch for you; you’ll need to update the way you structure your documents:

// Going from this:
{
    "id": 1,
    "name": "book 1",
    "categories": {
      "115": "poetry" 
    }
  }
// To this:
{
    "id": 1,
    "name": "book 1",
    "categories": ["poetry"]
  }

Then your filter would look like that: { "filter": "categories = poetry" }

[mhgolij]

Hey Again!

I have Just one error in my failed statuses and it is :

Error Message:
The primary key inference failed as the engine found 3 fields ending with `id` in their names: 'id' and 'attributes_id'. Please specify the primary key manually using the `primaryKey` query parameter.

This is odd because I set primaryKey = id when I was creating the index!

---

Stats Route Response:
{ "numberOfDocuments": 65190, "isIndexing": false, "fieldDistribution": { "attributes_id": 65190, "attributes_text": 65190, "canonical": 65190, "categories": 65190, "content": 65190, "created_at": 65190, "description": 65190, "galleries": 65190, "id": 65190, "is_active": 65190, "is_simple": 65190, "keywords": 65190, "low_cost_store_id": 65190, "meta": 65190, "name": 65190, "properties": 65190, "publishes_at": 65190, "sku": 65190, "slug": 65190, "stores": 65190, "thumb_url": 65190, "total_sale": 65190, "updated_at": 65190, "views": 65190, "weight": 65190 } }

---

Last Question:
What does this mean?
more than 65,536 attributes over all your documents

One of my Indexes stores 65190 objects, with each object having 26 fields. Some of these fields are objects themselves, containing at least 5 additional fields.
Another index stores 39075 objects, with each object having 10 fields.

[irevoire]

Hello @mhgolij, thanks a lot for your answer.
After your comment, I dug up the code and found out this was an ancient bug (more than two years old).
That’s why you did not see any error message in your task queue; sorry 😖
I opened an issue about it: #4464

---

I rushed my explanation a bit, sorry, so two things:

1. Every index needs to store the list of all the fields ever encountered and associate them with a 16-bit number.
2. A « field » is created for every value in your object. { "id": 12, "categories": { "151": "poetry" } } will count 3 fields, one for id, one for categories, and one for categories.151.

Another index stores 39075 objects, with each object having 10 fields.

This limitation is not shared between indexes.
Each index stores its own list of fields.

---

As a general idea, with meilisearch, we expect your fields to be pretty static and only the values changing.

Let me know if something is not clear, and thanks again for digging up the problem with me!

[mhgolij]

Hello @irevoire
Because I'm using two methods for search (MySQL and Meilisearch), I can't change the data structure, and the categories array must be changed sequentially in the same way.
Therefore, in this case, your suggestion is that I should not use Meilisearch?

[irevoire]

Hey @mhgolij,

Maybe you could create a view of your table formatted for meilisearch? See https://www.w3schools.com/mysql/mysql_view.asp

[mhgolij]

Thanks for your help❤️