How To Remove Duplicates From List In Python?

Python list data type provides the ability to store multiple items in a single variable or structure. A Python List can store the same or different types of items like integer, string, float, object, list, etc. While storing multiple items duplicate items can be also stored. If you need distinct values you should remove duplicates from the List. In this tutorial, we will examine how to remove duplicate items from a Python List. There is no official method in order to remove duplicates from a list but there are different ways to easily implement duplicate item removal from the list.

  • not in Keyword
  • List Comprehension
  • Set Conversion
  • List Comprehension and Enumeration
  • Ordered Dictionary
How To Remove Duplicates From List In Python?
How To Remove Duplicates From List In Python?

Remove Duplicates Using not in Keyword

The first method to find and remove duplicate items is using the not in keyword where the list items will be copied into a list and existing items will be not copied by checking with the, not in keyword. In the following example, the mylist contains duplicate items and the duplicate items will be removed by copying into newlist.

mylist = [1,2,3,4,5,6,2,4,5]

newlist = []

for i in mylist:
   if i not in newlist:
      newlist.append(i)

print(newlist)
Remove Duplicates Using not in Keyword

Remove Duplicates with List Comprehension

List comprehension is a similar method to the not in keyword where the iteration done in a single statement in a single line. There is no extra code but the for statement will be used in a single line.

mylist = [1,2,3,4,5,6,2,4,5]

newlist = []

[newlist.append(i) for i in mylist if i not in newlist]

print(newlist)
Remove Duplicates with List Comprehension

Remove Duplicates using Set

Python Set is an implementation of the mathematical sets where a set can not contain duplicate items. So Python set can be used to store list items where duplicate items will be removed. First, the list which contains duplicate items will be converted into a set and converted into a list back. One drawback of the method is the original order of the items will be lost because of the conversion and items will be ordered accordin to their value.

mylist = [1,2,3,4,5,6,2,4,5]

//Convert list into set where duplicate items will be automatically removed
myset = set(mylist)

//Convert set back to the list type
mylist = list(myset)

In previous example we have converted a list into set in one statement and converted set into list in another statement. We can also use a single statement where from list to set and set to list conversion can be done.

mylist = [1,2,3,4,5,6,2,4,5]

//Convert to set and then list again
mylist = list(set(mylist))

Remove Duplicates using List Comprehension and Enumeration

List comprehension is a method where list items are enumerated and already existing items will not add to the new list. The enumerate() function will be used to enumerate in the original list.

mylist = [1,2,3,4,5,6,2,4,5]

newlist = [a for n, a in enumerate(mylist) if a not in mylist[:n]]
Remove Duplicates using List Comprehension and Enumeration

Remove Duplicates using Ordered Dictionary

Dictionary is another popular data type or structure which stores key and value pairs. Even this structure is not the same as the list we can use the collections.OrderedDict.fromkeys() function where List items will be stored and duplicates are removed as keys in a dictionary.

from collections import OrderedDict

mylist = [1,2,3,4,5,6,2,4,5]

newlist = list(OrderedDict.fromkeys(mylist))

print(newlist)
Remove Duplicates using Ordered Dictionary

Leave a Comment