[How to] Retrieve all collections through multithread without duplication

Great to see you in the Community @ZhengTzerLee! Thank you for taking the time to ask your question. Using the information provided, this should be doable. Using /api/v4/collections/?limit=1&page={number} should loop through and give all different collections. If it's not working for you, it may be best to submit a support ticket for them to take a look. I can submit a ticket on your behalf if that’s helpful. 

Hi Ryanne

I try to submit the ticket myself, but no luck to identify the portal,

kindly raise the ticket on my behalf, thanks Ryanne!

issue background

tried to retrieve all collections through API in python with multi thread, but result either fall short (less than expected) or appear duplication records

what i tried

1st API call to obtain the total of collection need to retrieve

url = f"{base_url}?limit=1&page=1&count=1"

2nd API call

url = f"{base_url}?page={page}&limit=50&orderBy=dateCreated desc"

script message : 1st run result

INFO:root:Script started at: 2024-07-16 15:56:19.754188INFO:root:Total results: 2865, Total pages: 58100%|██████████| 58/58 [01:57<00:00,  2.02s/it]INFO:root:Progress end, total 2371 unique recordsINFO:root:Summary Report:INFO:root:Total records retrieved: 2371INFO:root:Unique records: 2371

script message : 2nd run result

INFO:root:Script started at: 2024-07-16 16:06:30.200857INFO:root:Total results: 2865, Total pages: 58100%|██████████| 58/58 [02:00<00:00,  2.07s/it]INFO:root:Progress end, total 2415 unique recordsINFO:root:Summary Report:INFO:root:Total records retrieved: 2415INFO:root:Unique records: 2415

notice there were 2865 collections, but each run will result different total


another method that i tried

use the total as page to loop through where each call is limit to 1, this approach will cause duplication

what works

not using multithread, let the script call 1 by 1 then no issue, this shouldn’t be

please check and let me know which part i did wrongly

thanks

Joanthan

Thank you for the additional info @ZhengTzerLee! I can confirm this has been sent to support as a ticket.