funboost-spider-crawling

Decorate a plain Python function with @boost(BoosterParams(queue_name=..., broker_kind=BrokerEnum.REDIS_ACK_ABLE, qps=5)), then call func.push(...) to dispatch tasks and func.consume() to start workers. Chain queues by pushing from one boosted function into another.

funspider provides SQLModel-based SpiderItem ORM with typed models and sync/async httpx clients, suiting team projects with managed schemas. boost_spider offers a requests-compatible RequestClient with proxy failover and dict-based DatasetSink storage, suiting minimal quick scripts.

Funboost schedules whole functions instead of Request objects, so you write linear code with any HTTP library rather than callback chains. It adds precise QPS control, multi-process concurrency, function-level retries, and Redis-based triggering, while Scrapy enforces Twisted and a fixed project layout.

Yes. Set concurrent_mode=ConcurrentModeEnum.ASYNC, define the consumer as async def, and use AsyncSpiderClient with await client.get(...). Tasks must be published with await func.aio_push(...) and items stored via aio_upsert().

Enable do_task_filtering=True in BoosterParams and set task_filtering_expire_seconds. Funboost deduplicates by function arguments in Redis, so identical pushed parameters are skipped within the expiration window, ignoring noisy URL query parameters.

Funboost retries at the function level: any exception raised inside the function triggers a retry up to max_retry_times. If a 200 response contains a captcha or empty data, raise an exception explicitly to force a retry, unlike Scrapy which only retries network failures.