Joining, Merging and Concatenation of DataFrames

Pandas lets us combine the data of two DataFrames into one. Although this section's title names joining, merging and concatenation together, the textbook (Reprint 2026-27) works these out through a single operation — appending one DataFrame to another. …

The pandas.DataFrame.append() method is used to merge two DataFrames: it appends the rows of the second DataFrame at the end of the first. Any columns present in the second DataFrame but not in the first are simply added as new columns to the result.

The textbook demonstrates this with two deliberately mismatched DataFrames. dFrame1 has columns C1, C2, C3 and rows R1, R2, R3 (with missing values filled as NaN, since its rows were given with different numbers of values); dFrame2 has columns C2, C5 and rows R4, R2, R5:

>>> dFrame1 = pd.DataFrame([[1, 2, 3], [4, 5], [6]],
...                        columns=['C1', 'C2', 'C3'],
...                        index=['R1', 'R2', 'R3'])
>>> dFrame1
    C1   C2   C3
R1   1  2.0  3.0
R2   4  5.0  NaN
R3   6  NaN  NaN

>>> dFrame2 = pd.DataFrame([[10, 20], [30], [40, 50]],
...                        columns=['C2', 'C5'],
...                        index=['R4', 'R2', 'R5'])
>>> dFrame2
    C2    C5
R4  10  20.0
R2  30   NaN
R5  40  50.0

Appending dFrame2 to dFrame1 stacks dFrame2's rows below dFrame1's; column C5 (which dFrame1 did not have) appears as a new column, and every cell with no value becomes NaN:

>>> dFrame1 = dFrame1.append(dFrame2)
>>> dFrame1
     C1    C2   C3    C5
R1  1.0   2.0  3.0   NaN
R2  4.0   5.0  NaN   NaN
R3  6.0   NaN  NaN   NaN
R4  NaN  10.0  NaN  20.0
R2  NaN  30.0  NaN   NaN
R5  NaN  40.0  NaN  50.0

Order matters. If we instead append dFrame1 to dFrame2, the rows of dFrame2 come first and the rows of dFrame1 follow.

The sort parameter controls the column order of the result. With sort=True the column labels appear in sorted order; with sort=False they appear unsorted (the order in which they were encountered):

# append dFrame1 to dFrame2 with sorted columns
>>> dFrame2 = dFrame2.append(dFrame1, sort=True)
>>> dFrame2
     C1    C2   C3    C5
R4  NaN  10.0  NaN  20.0
R2  NaN  30.0  NaN   NaN
R5  NaN  40.0  NaN  50.0
R1  1.0   2.0  3.0   NaN
R2  4.0   5.0  NaN   NaN
R3  6.0   NaN  NaN   NaN

# append dFrame1 to dFrame2 with sort=False
>>> dFrame2 = dFrame2.append(dFrame1, sort=False)
>>> dFrame2
      C2    C5   C1   C3
R4  10.0  20.0  NaN  NaN
R2  30.0   NaN  NaN  NaN
R5  40.0  50.0  NaN  NaN
R1   2.0   NaN  1.0  3.0
R2   5.0   NaN  4.0  NaN
R3   NaN   NaN  6.0  NaN

With sort=False the result keeps dFrame2's columns (C2, C5) first and then adds dFrame1's remaining columns (C1, C3).

The verify_integrity parameter guards against duplicate row labels. Setting verify_integrity=True makes append() raise an error if the row labels are duplicated. By default verify_integrity=False — which is why, in the examples above, the duplicate row labelled R2 (present in both DataFrames) could be appended without complaint. …