indice_hogar anio mes directorio orden mujer nivel_educativo_cat trabajaįor better understanding, I am using an IV that is the ability of the most similar person according to the index and to personal characteristics. It is sort of a nearest neighbor match but without having a control or treatment group. For my analysis I need to match the most similar observations based on these variables. My home index variable is numerical (from 0 to 103) and the personal characteristics are either dummies or categorical variables. Is there any other way to remove this kind of duplicates or something else I should consider/pay attention to?įor my thesis I need to match observations based on an index variable that measures home conditions, personal variables such as age, gender, education, etc. Processed.append(max( for x in scores], key=len)) Scores = process.extract(lines, df, scorer=fuzz.token_set_ratio) Here's my code: from thefuzz import fuzz, processĭf = pd.read_csv("file.csv", dtype=str, lineterminator='\n') However, it deletes some relevant lines or removes some of the duplicates only. I decided to use thefuzz to do this (which uses difflib). #letsrock We are joining the protest! I want to get rid of the last two lines while keeping the first one.We are joining the protest #protest #join.The file contains multiple duplicates that I want to get rid of while keeping only one original line. Even though I'm returned with a set of records, not able to tell if the records are duplicates or not.īelow are the different queries that i used:
DROP OBSERVATIONS IN STATA CODE
Since the table has no primary keys, I tried using a mixture of different fields like id, location code and amount, etc to find any recurring duplicates without mentioning the load date. The main issue was that I was not able to edit those files from beyond compare. After exporting the data to an excel file, I tried comparing the files using beyond compare but the files were being placed one after the other. I ran select queries for 17th, 18th, and 21st individually including 'X's id in the where condition. I'm going in with the assumption that there are duplicates and using queries or comparing excel files with data from different dates. I have been entering different queries to see if I could weed out the duplicates if there are any. Now, 'X' says that records from the 17th have also been uploaded on the 18th and 21st. Then the number of records is sent to 'X'.
The entries in 'InHouse' are uploaded to our databases to a table 'ABC'. 'X' sends files to us that are concatenated to a file 'InHouse' that we create. This is my first post and I apologize if I mess up any format.
0 kommentar(er)
