Writing functions in Python

Let's start with loading the libraries and generating some random data for the task

In [3]:

import numpy as np
import pandas as pd

In [4]:

# set seed for reproducibility, this will ensure that the random numbers generated are the same each time the code is run
np.random.seed(0)
# generate skewed variable
x = np.random.lognormal(mean=1, sigma=1, size=998)
# add two zeros for the exercise
x = np.concatenate([x, np.zeros(2)])
# into a data frame for ease of visual
df = pd.DataFrame({
    "x": x
})

df.head()

Out[4]:

	x
0	15.863999
1	4.055838
2	7.233608
3	25.556539
4	17.594001

In [ ]:

# lets see what that looks like
df['x'].hist(bins=30);

The variable I generated is very skewed so I need to transform it using log transformation before I use it in a model. Because I know I will be doing this in the future many time, I decide to wrap it up into an effective function, so I can reproduce this task in teh future easily.

* Let's start with a basic function definition

In [ ]:

def my_function(): # define inputs here
  # function task
  pass # what do you want it to return, you can use "return" here, pass statement is often used when developing, allowing you to define the structure first and implement details later

* Adjust the function to perform logarithmic transfromation of variable

In [ ]:

def transf_log( variable ): # one vector input
  
  variable_logged = np.log( variable ) # perform the log transformation

  pass # what do you want it to return, you can use "return" here, pass statement is often used when developing, allowing you to define the structure first and implement details later

Let's test if that works

In [ ]:

transf_log(x )

/var/folders/t5/57qdsp617xj8dd1by4td7fl40000gq/T/ipykernel_35779/1474781385.py:3: RuntimeWarning: divide by zero encountered in log variable_logged = np.log( variable ) # perform the log transformation

Looks like we have some zeros in the data, but $log(0) = inf$, meaning we cannot have zeros in the data before applying log transformation.

Let's expand the function for correction of the zeros.

You have two options,
1) Add a small constant to all values before applying the log transformation - the most common approach, shifts distribution by one point to the right. Remember this for any interpretations.
2) Replace zeros with NaN before applying the log transformation - this will result in loss of data points, so make sure the zeros are not important and it wont bias the results.
3) Replace zeros with a small constant before applying the log transformation, for example 0.001 - this is less common, but can be useful in some cases, sensitivity analysis might be necessary

In [ ]:

def transf_log( variable ): # define inputs here
  
  # handle non-positive values

  # 1. 
  variable = variable + 1 
  # 2. variable = np.where( variable = 0, np.nan, variable ) 
  # 3. variable = np.where( variable = 0, 0.01, variable ) 

  variable_logged = np.log( variable ) # perform the log transformation

  pass # one vector output

In [ ]:

transf_log(x )

That looks like it worked. Let's finish the function and apply it to data frame column

In [ ]:

def transf_log( variable ): # define inputs here
  
  # handle non-positive values

  # 1. 
  variable = variable + 1 
  # 2. variable = np.where( variable = 0, np.nan, variable ) 
  # 3. variable = np.where( variable = 0, 0.01, variable ) 

  variable_logged = np.log( variable ) # perform the log transformation

  return variable_logged # one vector output

In [ ]:

df['x_transformed'] = transf_log( df['x'] )
# lets see what that looks like
df['x_transformed'].hist(bins=30);

Amazing!
Why don't we expand this for the method too?

In [ ]:

def transf_log( variable, method_zero_correction = "add_one" ): # define inputs here
  """
  This is a space for docstring, you can describe what the function does, its inputs and outputs or any otehr details

    Description:    This function performs log transformation on a given variable, handling zero values based on the specified method.

    Inputs:             
        variable: one vector input
        method_zero_correction: method to handle zero values, options are "add_one", "replace_nan", "replace_small_value"
         "add_one" : adds 1 to all values before log transformation
         "replace_nan" : replaces zero values with NaN before log transformation
         "replace_small_value" : replaces zero values with a small value (0.01) before log transformation

    Outputs:
        variable_logged: log-transformed vector output
  """

  # we can add a condition using if-elif-else statements, be careful with indentation here         

  if method_zero_correction == "add_one":
    variable = variable + 1

  elif method_zero_correction == "replace_nan":
    variable = np.where( variable == 0, np.nan, variable )

  elif method_zero_correction == "replace_small_value":
    variable = np.where( variable == 0, 0.01, variable )       


  variable_logged = np.log( variable ) # perform the log transformation

  return variable_logged # one vector output

In [ ]:

df['x_transformed_replaced_with_nan'] = transf_log( variable = df['x'], method_zero_correction="replace_nan" )
# let's see what that looks like
df['x_transformed_replaced_with_nan'].hist(bins=30);

In [ ]:

df.info()

<class 'pandas.core.frame.DataFrame'> RangeIndex: 1000 entries, 0 to 999 Data columns (total 3 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 x 1000 non-null float64 1 x_transformed 1000 non-null float64 2 x_transformed_replaced_with_nan 998 non-null float64 dtypes: float64(3) memory usage: 23.6 KB