% Program taking raw x-ray and cryo data files mined from the RSC Protein 
% Data Bank and splitting them into separate files 
% Latest version 2023.10.16

% Notes on How to mine x-ray and cryo-EM data from the RSC PDB database 
% 2023.10.16
% 
% It has become apparent that even well into a given year, the deposits in the PDB from the previous year may not be complete and indeed this can continue to increase over several years. It is therefore recommended to do a full data mine using the below procedure, starting from the historically first deposit. Note also that in 2019, PDB had its first x-ray entry in 1972; in 2021, this entry seems to have disappeared and the first entry is at 1976. It is therefore important to keep an eye on these (somewhat inexplicable) changes. My approach is simply to take the most recent data. 
% This is the procedure I use to mine the data for the parsing program xrayCryoDataParse.m. 
% •	Go to https://www.rcsb.org/search 
% •	In the field “Structure Attributes”, select methods -> experimental methods and enter either x-ray diffraction or electron microscopy 
% •	Add a Subquery: Structure Details -> Deposition Date and enter “>” and the first date (e.g., 01.01.1976)
% •	Press SEARCH button 
% •	Click on Tabular Report-> Create Custom Report 
% •	Select deposit date, structure molecular weight, resolution IN THAT ORDER so columns are date, MW, resn in the CSV file then click on “run report” 
% •	Click on download CSV file. If there are more entries than 2500 for the search you have done, they are downloadable as chunks of 2500 at a time. In this case, you need to concatenate them (at the time of writing 2023.10.16, the entire x-ray data set required me to download 90 files of 2500 entries each!). Do this by: 
% o	Collect all the CSV files in a single folder 
% o	Open terminal in your mac 
% o	Change directory to this folder, e.g. cd Desktop/temp/
% o	Type in: cat *.csv >yourFilename.csv 
% •	Open this CSV file in Numbers (if you’re using a mac, otherwise, you have to work out the next bit yourself. Soz) 
% •	Go to the column header of the deposition date – this should be Column B 
% •	In the dropdown menu of this column, choose “sort ascending” 
% •	Click on first date entry (it should be the oldest) then scroll to bottom row and click on the last resolution entry while holding down shift key – this should highlight all three columns. Press cmd-C (copy)
% •	Open a new file in TexShop and press cmd-V (paste) – all the data should now be in the tex file. 
% •	Now you need to remove all dashes in the dates (e.g. 2020-05-05 changes to 2020  05  05, that is, with tabs in between. Spaces don’t work as matlab then thinks it is a string element). To do this simply press cmd-F (find) and enter “–“ in the field “Find” and a tab in “Replace With”, then click on “Replace All”. To get the tab in the replace field, open a blank tex file, enter a tab, then copy-paste it into the field 
% •	Save the file with the appropriate name e.g., allXrayData.dat, or allCryoData.dat 
% •	There might be some rows with multiple resolutions separated by a comma. The way I resolve this is as follows: Open up TeXShop. In TeXShop preferences, select OgreKit Find panel (if you aren’t already using this, you will need to restart TeXShop). Then press ctrl-f, click on ‘regular expressions’ in the options, and enter a comma followed by a space then (.+) in find [i.e. type in: , (.+)], and in replace, type in nothing. Do this for all instances of a comma. Not very clean, but hey-ho 
% •	The relevant columns are 
% o	1: The year 
% o	4: The molecular weight in kDa 
% o	5: The resolution in Angstroms 


clear all; close all; 

% Input data file names have date stamps indicating when they were mined. 
% Adjust accordingly and put them in the subfolders xrayyears and cryoyears
inDataX = importdata('xrayyears/allXrayData_20231016.dat'); 
inDataC = importdata('cryoyears/allCryoData_20231016.dat'); 
% column 1 = year; column 4 = MW [kDa]; column 5 = resolution [AA]
dataSizeX = size(inDataX); % Number of x-ray data points x 5 columns
dataSizeC = size(inDataC); % Number of cryo data points x 5 columns

NdataX = dataSizeX(1);           % Number of x-ray data points 
NdataC = dataSizeC(1);           % Number of cryo data points 
yrX = inDataX(1:NdataX,1:1);     % Year x-ray structure was deposited 
MWX = inDataX(1:NdataX,4:4);     % Molecular weight of x-ray structure
resnX = inDataX(1:NdataX,5:5);   % Resolution in Angstroms of x-ray structure 
yrC = inDataC(1:NdataC,1:1);     % Year cryo structure was deposited 
MWC = inDataC(1:NdataC,4:4);     % Molecular weight of cryo structure
resnC = inDataC(1:NdataC,5:5);   % Resolution in Angstroms of cryo structure 

str1X = 'xrayyears/year';        % Created files in directory 'xrayyears' 
str1C = 'cryoyears/year';        % Created files in directory 'cryoyears' 
str3 = '.dat'; 

yearX = inDataX(1,1);   % First year in x-ray data file 
yearC = inDataC(1,1);   % First year in cryo data file 
str2X = num2str(yearX); 
str2C = num2str(yearC); 

% Now split x-ray data into files according to the deposition year
for i = 1:NdataX
    str2 = num2str(yearX); 
    yearFileName = strcat(str1X,str2,str3); 
    fileID = fopen(yearFileName,'a+');
    
    if (yrX(i) == yearX) 
        fprintf(fileID,'%10.2f %10.3f\n',MWX(i),resnX(i)); 
        fclose(fileID); 
    elseif (yrX(i) > yearX)
        yearX = yrX(i); 
        fclose(fileID); 
        str2 = num2str(yearX); 
        yearFileName = strcat(str1X,str2,str3); 
        fileID = fopen(yearFileName,'a+'); 
        fprintf(fileID,'%10.2f %10.3f\n',MWX(i),resnX(i)); 
        fclose(fileID); 
    end
end 

% Now split cryo data into files according to the deposition year
for i = 1:NdataC
    str2 = num2str(yearC); 
    yearFileName = strcat(str1C,str2,str3); 
    fileID = fopen(yearFileName,'a+');
    
    if (yrC(i) == yearC) 
        fprintf(fileID,'%10.2f %10.3f\n',MWC(i),resnC(i)); 
        fclose(fileID); 
    elseif (yrC(i) > yearC)
        yearC = yrC(i); 
        fclose(fileID); 
        str2 = num2str(yearC); 
        yearFileName = strcat(str1C,str2,str3); 
        fileID = fopen(yearFileName,'a+'); 
        fprintf(fileID,'%10.2f %10.3f\n',MWC(i),resnC(i)); 
        fclose(fileID); 
    end
end 