Marcus hsieh/tasks refactor #526
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit e494935.
…ieh/merlin into MarcusHsieh/task_server
+ Performance limit tests + Pytest performance markers
TWO LAYERS: 1. Task Distribution Layer (TaskServerInterface) 2. Worker Management Layer (MerlinWorkerHandler)
Integration Components - Kafka worker extended by worker interface - Handler for managing Kafka workers - WorkerHandlerFactory to register and support Kafka backend - --backend flag
- Replace direct Celery function calls with script references - Context-free execution enabling any message broker backend - Message size optimization: 75% reduction (427-442 bytes vs 1,715 bytes) - All messages under 1MB target for efficient network transmission - Add TaskScriptGenerator class for converting steps to executable scripts - Add MessageOptimizer for compact task message generation - Rename kafka_worker.py to kafka_worker_manager.py - Rename implementations/kafka_worker.py to kafka_task_consumer.py
…factory system Core Components: - Universal Task Factory (merlin/factories/) - Backend-agnostic task creation - Signature Adapters (merlin/adapters/) - Celery & Kafka backend support - Sample Expansion Optimization (merlin/optimization/) - Range-based batching - Compressed JSON Serialization (merlin/serialization/) - 63-85% compression Features: - 1,123,274 tasks/sec creation rate - 63-85% message size reduction via gzip+field optimization - Complete coordination patterns (group: group_id + partitions, chain: task dependencies, chord: group_id + callback tasks) - Pytest suite (+5 test files) Performance Results: - Simple tasks: 433B -> 157B (63.7% reduction) - Large tasks: 3466B -> 536B (84.5% reduction) - Sample processing: 158M+ samples/sec (data organization/division)
- merlin/common/tasks.py: +Universal Task support - merlin/coordination/: +Coordination pattern modules - merlin/task_servers/: +Celery server implementation - merlin/workers/: +Universal Task System worker integration
- Fix test_kafka_worker.py broken import of non-existent KafkaWorker class - Update imports to use actual KafkaTaskConsumer implementation - Fix kafka_server.py import paths to reference correct modules - Add compatibility methods to KafkaTaskConsumer for test interface - Fix test_integration.py broken StudyExecutor reference NOTE: - KafkaWorkerManager: Worker lifecycle management (merlin/workers/) - KafkaTaskConsumer: Actual task processing implementation (merlin/task_servers/) - KafkaTaskWorkerRuntime: Legacy embedded runtime for compatibility
- Fix mock path from module-level to correct kafka.KafkaConsumer - Update NameError references from KafkaWorker to KafkaTaskConsumer - Fix task_registry import path to merlin.execution.task_registry
- Fix _initialize_consumer to properly call consumer.subscribe() - Separate consumer creation from subscription for proper test mocking - Add merlin_control topic to all subscription calls
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.