Blog

In the realm of data science, where insights are drawn from complex computations and algorithms, logging is an indispensable tool. Proper logging not only helps track the workflow and debug issues but also enhances the transparency, reproducibility, and maintainability of data science projects. In this first installment of the series “Professionalize Data Science Codes,” we delve into the importance of proper logging and explore strategies to implement it effectively.

Why Logging?

Logging serves as a trail of breadcrumbs left by your code as it executes. It allows developers to:

1. Monitor Execution Flow: Understand how the code progresses and identify potential bottlenecks.
2. Debug Efficiently: Pinpoint the source of errors or unexpected behavior without interrupting the process flow.
3. Track Performance: Measure time taken by specific functions or components and assess the efficiency of algorithms.
4. Enhance Reproducibility: Maintain records of configurations, inputs, and outputs, making it easier to reproduce results.
5. Provide Transparency: Enable better collaboration by making the inner workings of the code comprehensible to others.

Without logging, troubleshooting becomes an exercise in frustration, particularly in long-running or complex computations.

Levels of Logging

Logging frameworks categorize logs into different levels, enabling selective verbosity:

1. DEBUG: Detailed information for debugging, typically verbose.
2. INFO: General information about the program’s execution.
3. WARNING: Indicators of potential problems or non-critical issues.
4. ERROR: Errors that prevent specific functionalities from executing.
5. CRITICAL: Severe errors that may cause the program to terminate.

Using the appropriate logging level ensures that the log files are both informative and manageable.

What is Loguru in Python?

Loguru is a modern and powerful logging library for Python that simplifies logging with a developer-friendly API. It eliminates the need for boilerplate code associated with Python’s built-in logging module. Key features of Loguru include:

• Ease of Use: Logging a message is as simple as calling logger.info(“Your message”).
• Customizable Outputs: Log messages can be routed to multiple destinations, such as files, consoles, or even databases.
• Structured Logging: Add contextual data to logs using .bind().
• Exception Handling: Automatically log uncaught exceptions with detailed stack traces.
• Dynamic Configuration: Adjust log formats, levels, and destinations dynamically.

Loguru has become a favorite among developers due to its flexibility and ease of integration.

Best Practices for Logging

Implementing effective logging involves following best practices to ensure logs are meaningful, concise, and actionable:

1. Use a Consistent Format

Logs should follow a standard format that includes relevant details such as timestamps, log levels, module names, and messages. Loguru allows customization of formats to suit specific needs. For example:

from loguru import logger

logger.add(

    "logs.log",

    format="{time:YYYY-MM-DD HH:mm:ss} | {level} | {message}",

    level="DEBUG",

)

2. Implement monitor_runtime Decorator

A monitor_runtime decorator can be used to log the execution time and performance of functions. For example:

from functools import wraps

import time

from loguru import logger

def monitor_runtime(step_name=None):

    def wrapper(func):

        @wraps(func)

        def timed(*args, **kwargs):

            actual_step_name = step_name if step_name else func.__name__

            logger.info(f"Starting step: {actual_step_name}")

            start_time = time.time()

            result = None

            try:

                result = func(*args, **kwargs)

            except Exception as e:

                logger.error(f"Step failed: {actual_step_name} with exception: {e}")

                raise

            finally:

                elapsed_time = time.time() - start_time

                logger.info(f"Finished step: {actual_step_name} | Execution Time: {elapsed_time:.2f} seconds")

            return result

        return timed

    return wrapper

Applying this decorator to a function provides detailed logs of its runtime:

@monitor_runtime("Data Loading")

def load_data(filepath):

    # Simulate data loading

    time.sleep(2)

    return {"data": "sample"}

load_data("data.csv")

3. Log Performance Metrics

Logging isn’t just for errors and warnings; it’s invaluable for tracking performance metrics like:

• Execution Time: Use the monitor_runtime decorator to measure durations of various steps in the pipeline.
• Model Training and Inference: Log key statistics such as training duration, model accuracy, and inference speed.
• Memory Usage: Track memory consumption for resource-intensive tasks.

Example:

@monitor_runtime("Model Training")

def train_model(model, data):

    accuracy = model.fit(data)

    logger.info(f"Model training completed with accuracy: {accuracy:.2f}")

    return model

4. Log to Files

Storing logs in files ensures they are available for post-mortem analysis. Loguru makes it simple to add file-based logging:

logger.add(“application.log”, level=”INFO”, rotation=”10 MB”, retention=”7 days”)

This configuration:

• Rotates logs once they exceed 10 MB.
• Retains logs for 7 days.

5. Log Contextual Information

Include contextual data in your logs to make them more informative. For instance:

logger.bind(user="john_doe", process_id=12345).info("Processing started")

This enhances traceability in distributed systems.

6. Avoid Logging Sensitive Information

Ensure that sensitive data like passwords, tokens, or personal information is redacted or excluded from logs.

7. Maintain Log Levels Appropriately

• Use DEBUG during development.
• Use INFO for general runtime information.
• Use ERROR or CRITICAL for production issues.

8. Test Your Logging Setup

Regularly review logs to ensure they capture the intended information. Misconfigured logging can lead to noisy or incomplete logs.

Logging Performance of Machine Learning Models

Performance tracking is a critical aspect of machine learning workflows. Logging key metrics ensures transparency and enables optimization. Here’s how to log performance effectively:

Logging Training Metrics

During training, log metrics like:

• Execution Time: How long the training process took.
• Accuracy: The model’s accuracy on the validation dataset.
• Loss: The loss function values across epochs.

Example:

@monitor_runtime("Model Training")

def train_model(model, train_data, val_data):

    history = model.fit(train_data, validation_data=val_data, epochs=10)

    accuracy = history.history["val_accuracy"][-1]

    logger.info(f"Training completed. Validation Accuracy: {accuracy:.2f}")

    return model

Logging Inference Metrics

For inference, log metrics such as:

• Latency: Time taken to make predictions.
• Throughput: Number of samples processed per second.

Example:

@monitor_runtime("Model Inference")

def run_inference(model, test_data):

    start_time = time.time()

    predictions = model.predict(test_data)

    latency = time.time() - start_time

    logger.info(f"Inference completed in {latency:.2f} seconds")

    return predictions

Monitoring Deployment Performance

For deployed models, track metrics such as:

• Request Latency: Time taken to respond to API calls.
• Error Rates: Percentage of failed predictions.
• Resource Utilization: CPU and memory usage.

Integrating logs with monitoring tools like Prometheus or Grafana provides real-time insights.

Advanced Logging Techniques

Asynchronous Logging

Use asynchronous logging to prevent I/O operations from blocking your main process. For example, Loguru supports threading:

logger.add("async_logs.log", enqueue=True)

Distributed Logging

In distributed systems, centralize logs using tools like ELK Stack (Elasticsearch, Logstash, and Kibana) or Fluentd.

Structured Logging

Use JSON or other structured formats to make logs machine-readable and easier to analyze.

logger.add("structured_logs.json", serialize=True)

Conclusion

Proper logging is a cornerstone of professional data science code. By following best practices and leveraging powerful tools like Loguru, developers can create robust, maintainable, and transparent systems. From tracking execution flow to monitoring performance metrics, logging ensures that your code is ready for both debugging and scaling.