Category: Python

Sometimes, we want to detect and exclude outliers in Pandas data frame with Python.

In this article, we’ll look at how to detect and exclude outliers in Pandas data frame with Python.

How to detect and exclude outliers in Pandas data frame with Python?

To detect and exclude outliers in Pandas data frame with Python, we can use NumPy to return a new DataFrame that has values within 3 standard deviations from the mean.

To do this, we can write:

import pandas as pd
import numpy as np

df = pd.DataFrame({'Data':np.random.normal(size=200)})
new_df = df[np.abs(df.Data-df.Data.mean()) <= (3*df.Data.std())]
print(new_df)

We create a Pandas DataFrame with a normal distribution with sample size 200 with np.random.normal.

Then we pick the values that are within 3 standard deviations from the mean with df[np.abs(df.Data-df.Data.mean()) <= (3*df.Data.std())].

And we assign the returned DataFrame to new_df.

Therefore, new_df is something like:

         Data
0    0.300805
1   -0.474140
2   -0.326278
3    0.566571
4   -1.391077
..        ...
195  0.500637
196  0.341858
197 -1.058419
198 -0.565920
199 -1.008344

[200 rows x 1 columns]

according to print.

Conclusion

To detect and exclude outliers in Pandas data frame with Python, we can use NumPy to return a new DataFrame that has values within 3 standard deviations from the mean.

Sometimes, we want to list a directory tree in Python.

In this article, we’ll look at how to list a directory tree in Python.

How to list a directory tree in Python?

To list a directory tree in Python, we can use the os.walk method.

For instance, we write:

import os

for dirname, dirnames, filenames in os.walk('.'):
    for subdirname in dirnames:
        print(os.path.join(dirname, subdirname))

    for filename in filenames:
        print(os.path.join(dirname, filename))

We call os.walk with the root path string to return an iterator with tuples with dirname, dirnames, and filenames.

Then we can loop through dirnames and filenames and get the subdirectories and files in each directory respectively.

We call os.path.join to get the full subdirectory and file paths respectively.

Therefore, we get something like:

./.upm
./pyproject.toml
./poetry.lock
./test.csv
./art.png
./.breakpoints
./main.py
./.upm/store.json

from the print calls.

Conclusion

To list a directory tree in Python, we can use the os.walk method.

Sometimes, we want to convert a string to lower case in Python.

In this article, we’ll look at how to convert a string to lower case in Python.

How to convert a string to lower case in Python?

To convert a string to lower case in Python, we can call the string’s lower method.

For instance, we write:

s = "Kilometer"
print(s.lower())

We call s.lower to return "Kilometer" with all lower case characters.

Therefore, 'kilometer' is printed.

Conclusion

To convert a string to lower case in Python, we can call the string’s lower method.

Sometimes, we want to merge a list of dicts into a single dict with Python.

In this article, we’ll look at how to merge a list of dicts into a single dict with Python.

How to merge a list of dicts into a single dict with Python?

To merge a list of dicts into a single dict with Python, we can use reduce function from the functools module.

For instance, we write:

from functools import reduce

list_of_dicts = [{'a': 1}, {'b': 2}, {'c': 1}, {'d': 2}]
d = reduce(lambda a, b: dict(a, **b), list_of_dicts)
print(d)

We call reduce with a function that merges dictionary a with the entries in b and return it.

a and b are both entries in list_of_dicts.

The merging is done by unpacking the entries in b and putting it into a new dictionary with a and returning it.

Therefore, d is {'a': 1, 'b': 2, 'c': 1, 'd': 2}.

Conclusion

To merge a list of dicts into a single dict with Python, we can use reduce function from the functools module.

Sometimes, we want to convert a hexadecimal string to byte array in Python.

In this article, we’ll look at how to convert a hexadecimal string to byte array in Python.

How to convert a hexadecimal string to byte array in Python?

To convert a hexadecimal string to byte array in Python, we can use the bytes.fromhex method.

For instance, we write:

hex_string = "deadbeef"
s = bytes.fromhex(hex_string)
print(s)

We define the hex_string hex string.

Then we call bytes.fromhex with it as the argument and assign the returned byte array to s.

Therefore, s is b'\xde\xad\xbe\xef'.

Conclusion

To convert a hexadecimal string to byte array in Python, we can use the bytes.fromhex method.