Advanced Search

Remove duplicates from a list

/ Published in: Django
Save to your folder(s)
Not only is it really really fast; it's also order preserving and supports an optional transform function

Expand | Embed | Plain Text

Copy this code and paste it in your HTML 
  1. def remove_duplicates(seq, idfun=None):  
  2.     # order preserving 
  3.     if idfun is None: 
  4.         def idfun(x): return x 
  5.     seen = {} 
  6.     result = [] 
  7.     for item in seq: 
  8.         marker = idfun(item) 
  9.         # in old Python versions: 
  10.         # if seen.has_key(marker) 
  11.         # but in new ones: 
  12.         if marker in seen: continue 
  13.         seen[marker] = 1 
  14.         result.append(item) 
  15.     return result
  16.  
  17.  
  18.  
  19. >>> a=list('ABeeE') 
  20. >>> f5(a) 
  21. ['A','B','e','E'] 
  22. >>> f5(a, lambda x: x.lower()) 
  23. ['A','B','e']

URL: http://www.peterbe.com/plog/uniqifiers-benchmark

Report this snippet


Comments

RSS Icon Subscribe to comments

You need to login to post a comment.