How Complete Are Immunization Registries?
The Philadelphia Story
Maureen S. Kolasa, RN, MPH; Andrew P. Chilkatowsky, BA;
Kevin R. Clarke, MD; James P. Lutz, MPA
Objective.–To assess accuracy and completeness of Philadel-
phia, Pa, registry data among children served by providers in
areas at risk for underimmunization.
Methods.–Philadelphia’s Department of Public Health selected a
simple random sample of 45 children age 19 –35 months (or all
children age 19 –35 months if there were ⬍45 children in the
practice) from each of 30 private practices receiving govern-
ment-funded vaccine and located in zip codes where children are
at risk for underimmunization. Chart and registry data were
compared with determine the proportion of children missing
from the registry and assess differences in immunization
coverage.
Results.–Of 620 children reviewed, 567 (92%) were in the
registry. Significant differences (P ⬍ .05) were observed in
immunization coverage for 4 diphtheria-tetanus-acellular pertus-
sis vaccinations, 3 polio vaccinations, 1 measles-mumps-rubella
vaccination, and 3 Haemophilus influenzae type b vaccinations
between the chart (80% coverage) and registry (62% coverage).
Providers submitting electronic medical records or directly trans-
ferring electronic data to the registry had significantly more
children in the registry and higher registry-reported immuniza-
tion coverage than those whose data were entered from billing
records or log forms. All practice types experienced difficulties
in transferring complete data to the registry.
Conclusions.–Although 92% of study children were in the reg-
istry, immunization coverage was significantly lower when reg-
istry data were compared with chart data. Because electronic
medical records and direct electronic data transfer resulted in
more complete registry data, these methods should be encour-
aged in linking providers with immunization registries.
KEY WORDS: health services; immunization; registries; vac-
cination
Ambulatory Pediatrics 2006;6:21–24
H
ealthcare providers and parents face difficult chal-
lenges in keeping children up to date (UTD) for
immunizations. The immunization schedule is in-
creasingly complex. Currently, 22 immunizations are rec-
ommended for infants by 18 months of age, and this
number could triple by 2020.
1,2
Among children ⬍35
months of age, 22% have seen 2 or more immunization
providers, and almost a quarter of immunizations were not
administered by the last provider of care.
3,4
One reason for
this is that families frequently relocate and change medical
providers.
5,6
These factors make it difficult for parents and
providers to ensure that children are vaccinated appropri-
ately.
7
More than 25% of children age 19 –35 months are
not UTD for all recommended vaccines, and 10.6% re-
ceive at least one unnecessary dose of vaccine.
8,9
Immunization registries help to address the challenges
of keeping children UTD for vaccinations. Registries pro-
vide computerized systems for maintaining information
from multiple providers on children’s vaccinations within
a geographic area.
5,10
Providers can use registries to de-
termine children’s vaccination histories and to determine
vaccinations due. The Healthy People 2010 objectives
include enrollment of 95% of children age 5 years or
younger in a confidential population-based immunization
registry.
11,12
The National Vaccine Advisory Committee
recommends that immunization registries involving both
public and private providers be developed in each
state.
13,14
In response to these recommendations, comput-
erized immunization registries are being implemented
throughout the United States, with 44 states currently
operating regional or statewide systems.
15,16
The completeness of immunization registry data di-
rectly impacts registry usefulness. Health care providers
should be able to quickly check the registry to determine
the need for immunizations during preventative or acute
care visits.
17
A major obstacle to meeting this goal is lack
of widespread provider participation in registries, resulting
in incomplete records.
5,18
Incomplete immunization
records further discourage private provider participation
and use of registry data.
2,5,7,14
To identify barriers to the
usefulness of and participation in the Philadelphia (Pa)
KIDS Immunization Registry, the Philadelphia Depart-
ment of Public Health evaluated the completeness of reg-
istry data by provider type and method of data entry into
the registry.
METHODS
The Philadelphia Department of Public Health (PDPH)
assessed immunization records at 30 private immunization
providers from January to July 2003. These 30 providers
From the National Immunization Program, Centers for Disease Con-
trol and Prevention, Atlanta, Ga (Ms Kolasa); National Immunization
Program, Centers for Disease Control and Prevention, Philadelphia, Pa
(Mr Chilkatowsky, Mr Lutz); and University of California, San Fran-
cisco, Calif (Dr Clarke).
The findings and conclusions in this report are those of the authors and
do not necessarily represent the views of the funding agency.
Address correspondence to Maureen S. Kolasa, 1600 Clifton Rd NE,
Received for publication May 24, 2005; accepted August 15, 2005.
AMBULATORY PEDIATRICS Volume 6, Number 1
Copyright © 2006 by Ambulatory Pediatric Association January–February 2006
21
were randomly selected from the approximately 90 private
providers serving children 19 –35 months of age and lo-
cated in north and west Philadelphia, the areas of Phila-
delphia at highest risk for underimmunization among
preschool children. The 30 providers serve about 29% of
the children in the City of Philadelphia and receive gov-
ernment-funded vaccine through the Vaccines for Chil-
dren program.
Children were considered eligible for the study if they
were 19 –35 months of age and had received at least one
immunization at the provider’s office. For providers with
ⱕ45 children aged 19 –35 months on the date of the
provider assessment, all children were included in the
immunization assessment. For providers with ⬎45 chil-
dren, a random sample of 45 children was taken. Among
the 30 providers, charts of 620 children 19 –35 months of
age were assessed. PDPH staff entered immunization in-
formation for these children into an immunization soft-
ware program (Clinical Assessment Software Application)
and then assessed each child’s UTD immunization status
according to the recommended vaccination schedule.
19,20
Philadelphia’s KIDS Immunization Registry was then
searched for data on every child included in the chart
assessment. Immunization providers enter children into
the registry. Children missing from the registry were cat-
egorized by the type of provider administering their im-
munizations and by provider’s method of inputting data
into the registry. Provider types examined were 7 pediatric
practices, 19 family practices, 3 hospital-based providers,
and 1 shelter that administers immunizations. Methods of
inputting data into the registry were log forms (13 provid-
ers, including 5 pediatric practices and 8 family practices),
paper billing records (7 providers, including 1 pediatric
practice and 6 family practices), electronic billing records
(6 providers, including 1 hospital-based provider, 1 pedi-
atric practice, and 4 family practices), electronic medical
records (2 providers, both hospital-based providers) and
direct electronic transfer of data (2 providers, including 1
family practice and 1 shelter). Log forms were defined as
forms manually generated at the clinic and sent to PDPH
for manual entry into the registry. Paper copies of billing
records also were sent to PDPH for manual entry into the
registry. Electronic billing and medical records were either
transferred into a secure web file repository system or
submitted via a disk to PDPH and uploaded into the
registry. Direct electronic transfer of data meant that data
automatically went from the provider’s computer system
directly into the registry.
Immunization coverage levels based on chart data and
registry data were compared for each provider. When
calculating coverage based on registry data, vaccinations
administered and entered into the registry by any provider
were included in the analysis. Vaccinations administered
after the date of the provider chart assessment were ex-
cluded from the analysis. Coverage level calculations for
the chart and registry were based on the sample selected at
the provider’s office. Analysis was stratified by provider
type and method of data entry into the registry. Children
ⱖ19 months of age were assessed because these children
should already have received all preschool immunizations.
To measure agreement between UTD status as calculated
based on the chart and registry, we calculated
statistics
and the McNemar test for marginal homogeneity. Calcu-
lations were performed by SAS software, version 8.2 for
Windows (SAS Institute, Cary, NC).
21
In reporting immunization coverage throughout this ar-
ticle, immunization coverage for 4 doses of diphtheria,
tetanus toxoids, and pertussis vaccine, 3 doses of poliovi-
rus vaccine, 1 dose of measles-mumps-rubella vaccine,
and 3 doses Haemophilus influenzae type b vaccine are
grouped together (4:3:1:3) because this is the widely ac-
cepted recommended vaccine series.
8
Coverage rates for
the varicella vaccine are reported separately. Hepatitis b
vaccine was not assessed because of inconsistencies
among birthing hospitals in reporting the first dose to the
registry. Pneumococcal and influenza vaccination were not
assessed because they were not recommended vaccines for
all children age 19 –35 months during the entire study
period.
RESULTS
Registry Participation
Of the 620 children’s charts assessed, 567 children
(92%) were found in the immunization registry and 473
(76%) had data input into the registry by at least one of the
30 study providers. The remaining 94 children were in the
registry only because another immunization provider had
entered their data, indicating that the study provider had
failed to submit data on the immunization(s) that they had
administered to these 94 children. Providers submitting
data to the registry via direct electronic entry or electronic
medical records had significantly (P ⬍ .05) more patients
(121/121, 100%) in the registry than providers sending
electronic billing data (88/96, 92%), paper billing data
(88/101, 87%), or paper log forms (270/302, 89%) to the
registry. Hospital-based practices had significantly (P ⬍
.05) more of their patients in the registry (123/127, 97%)
than pediatric practices (200/223, 90%) or family practices
(240/266, 90%).
Immunization Coverage
Combining data for all 30 providers, UTD immuniza-
tion coverage for 4:3:1:3 for children 19 –35 months of age
was 80% according to the chart and 62% according to the
registry (P ⬍ .05). For all antigens, antigen-specific cov-
erage based on the chart was significantly higher (P ⬍ .05)
than when based on the registry. Coverage at 19 months of
age followed similar trends as coverage at 19 –35 months
of age.
When immunization coverage was compared by method
of registry data entry (Figure 1), only direct entry of
electronic data resulted in no difference between chart-
based and registry-based coverage levels, with both show-
ing immunization coverage of 87% among children 19–35
months of age. With all other methods of entering data into
the registry, chart-based coverage was significantly greater
than registry-based coverage (P ⬍ .05). UTD coverage
was 84% chart versus 58% registry (26 percentage point
AMBULATORY PEDIATRICS22 Kolasa et al
difference,
⫽ 0.42) when manual log forms were used,
75% chart versus 59% registry (16 percentage point dif-
ference,
⫽ 0.65) when paper billing records were used
and 70% chart versus 53% registry (17 percentage point
difference,
⫽ 0.66) when electronic billing records were
used. Other than direct entry, use of electronic medical
records to enter data into the registry provided the highest
level of registry agreement with the chart, with chart UTD
coverage of 79% chart and registry UTD coverage of 76%
(3 percentage point difference,
⫽ 0.72).
When chart and registry data were compared by practice
type (Figure 2), hospital-based practices showed immuni-
zation coverage for 4:3:1:3 among children age 19 –35
months to be 79% based on the chart and 72% based on
the registry (7 percentage point difference,
⫽ 0.81).
Among pediatric practices, children had coverage of 87%
according to the chart and 61% according to the registry
(26 percentage point difference,
⫽ 0.37). Among family
practices, children aged 19 –35 months had chart coverage
of 74% and registry coverage of 58% (16 percentage point
difference,
⫽ 0.63).
DISCUSSION
Most children (92%) 19 –35 months of age sampled
from the 30 private providers in this evaluation were in
Philadelphia’s immunization registry. However, the pro-
viders in the study entered no information into the registry
for almost a quarter of the children, even though they had
administered at least one immunization to these children.
These children were in the registry only because other
providers had input their information.
Electronic medical records and systems designed to
directly transfer data to the registry were the most effec-
tive methods of entering information in the registry. Elec-
tronic and paper billing records, along with paper log
forms, were significantly associated with fewer children
ever entered in the registry and lower UTD coverage
estimates. Immunization providers should therefore con-
sider adopting electronic medical records or systems that
directly transfer data into immunization registries. How-
ever, adoption of an electronic medical record system does
not ensure an effective interface with all immunization
registries. The technical requirements needed to transfer
data to the local immunization registry should be one
consideration in purchasing or designing an electronic
medical record system for any practice.
Hospital-based practices were more thorough than pe-
diatric and family practices, both in initially entering their
clients into the registry and keeping the registry UTD with
additional immunizations. Two of the 3 hospital-based
practices also used electronic medical records, which were
also significantly associated with entering more children
into the registry and inputting more immunizations into
the registry. Because the evaluation was limited in the
numbers of hospital-based practices and the number of
practices that used electronic medical records, it is unclear
whether being a hospital-based practice or a practice with
electronic medical records or a combination of both was
responsible for the high rates of data entry into the registry
by these practices.
Although the pediatric and family practices in our eval-
uation did not enter as many of their patients into the
registry as the hospital-based clinics, they were still able to
enter most (90%) of them. However, pediatric and family
practices experienced more difficulty consistently entering
their patient’s immunization records into the registry and,
as a result, the records of these children were less complete
than records of children visiting hospital-based clinics.
Because this problem was as apparent for children aged
19 –35 months as for children just 19 months of age, time
did not appear to assist these providers in getting their data
into the registry. Again, it is unclear whether the type of
provider practice or the method of data entry was respon-
sible for the less complete records in these groups.
Along with the difficulty in separating type of provider
from data entry method, this study has additional limita-
tions. The study took place in a single urban area and the
population is not generalizable. Philadelphia’s immuniza-
84
75
70
79
87
58
59
53
76
87
0
10
20
30
40
50
60
70
80
90
100
Manual Logs Manual
Billing
Electronic
Billing
Elec Med
Record
Direct Entry
chart registry
%
UTD
4313
*
*
N=302
Kappa .42
95% LCL .33
95% UCL .51
N=96
Kappa .66
95% LCL .51
95% UCL .80
N=90
Kappa .72
95% LCL .56
95% UCL .87
N=31
Kappa 1
95%LCL 1
95%UCL 1
N=101
Kappa .65
95% LCL .50
95% UCL .80
*
*
* = p<.05; LCL = Lower Confidence Limit; UCL = Upper Confidence Limit;
UTD = Up-to-Date for Immunizations; 4313 = 4DTaP, 3polio, 1MMR, 3Hib vaccinations
Figure 1. 4:3:1:3 (4 doses of diphtheria, tetanus toxoids, and pertussis
vaccine, 3 doses of poliovirus vaccine, 1 dose of measles-mumps-rubella
vaccine, and 3 doses Haemophilus influenzae type b vaccine) immuniza-
tion coverage by data entry method.
80
79
87
74
62
72
61
58
0
10
20
30
40
50
60
70
80
90
100
Overall Hospital-Based Pediatric Practice Family Practice
Chart Registry
%
UTD
4313
*
*
*
N=620
Kappa .58
95% LCL .51
95%UCL .64
N=123
Kappa .81
95% LCL .68
95% UCL..92
N=223
Kappa .37
95%LCL .27
95%UCL .48
N=266
Kappa .63
95%LCL .54
95%UCL .72
*
*
= p<.05; Shelter (n=8) not shown; LCL = Lower Confidence Limit; UCL = Upper Confidence Limit;
UTD = Up-to-Date for Immunizations; 4313 = 4DTaP, 3polio, 1MMR, 3Hib vaccinations
Figure 2. 4:3:1:3 (4 doses of diphtheria, tetanus toxoids, and pertussis
vaccine, 3 doses of poliovirus vaccine, 1 dose of measles-mumps-rubella
vaccine, and 3 doses Haemophilus influenzae type b vaccine) immuniza-
tion coverage by provider type.
AMBULATORY PEDIATRICS How Complete Are Immunization Registries? 23
tion registry is uniquely designed for the City of Philadel-
phia. These findings are specific to the Philadelphia
immunization registry and may not be generalizable to
other registries. However, this study could be used to
trigger similar studies of other immunization registries.
The immunization coverage levels based on the chart
assessment in this study were equivalent to those found by
the National Immunization Survey during the same year.
In 2003, the survey found that 80% (⫾5.6%) of children
age 19 –35 months in Philadelphia County were UTD for
4:3:1:3.
8,22
In our sample, overall coverage for 4:3:1:3 was
also 80% when calculated with the chart information. This
similarity in findings is an indication that the physicians
and children sampled in our study were reasonably com-
parable to other physicians and children in Philadelphia
County in 2003.
This study suggests that although providers are entering
most children and many immunizations into the Philadel-
phia immunization registry, improvement is needed before
data in the registry is as complete as data in patient charts.
In particular, pediatric and family practices need to im-
prove their methods of submitting data to the immuniza-
tion registry. Use of electronic medical records and direct
transfer of immunization data into the registry are effec-
tive methods of transferring this information and should be
considered by providers seeking to improve their input of
data into immunization registries. Because not all provid-
ers will be able to adopt electronic medical records or a
direct transfer method, further study is needed to identify
effective methods to improve the process of submitting
immunization records to registries that use paper and elec-
tronic billing records and log forms. Through improved
methods of data submission, the Philadelphia immuniza-
tion registry will be better able to serve providers in
determining the immunization needs of individual clients
and the public health system will be better able to assess
immunization coverage levels and identify children in
need of outreach in order to be brought UTD for immu-
nizations.
ACKNOWLEDGMENTS
We thank Tanya Jones and John O’Callahan of the City of Philadel-
phia Department of Public Health for their help with data collection.
REFERENCES
1. Centers for Disease Control and Prevention. Recommended child-
hood and adolescent immunization schedule—United States, 2005.
MMWR Morb Mortal Wkly Rep. 2005;53:Q1–Q3.
2. Linkins RW. Immunization registries: progress and challenges in
reaching the 2010 national objective. J Public Health Manage Pract.
2001;7:67–74.
3. Stokley S, Rodewald LE, Maes EF. The impact of record scatter-
ing on the measurement of immunization coverage. Pediatrics
2001;107:91–96.
4. Kolasa MS, Cherry JE, Chilkatowsky AP, Reyes DP, Lutz JP.
Practice-based electronic billing systems and their impact on immu-
nization registries. J Public Health Manage Practice. 2005;11(6):
484 – 492.
5. Centers for Disease Control and Prevention. Development of
community- and state-based immunization registries: CDC response
to a report from the National Vaccine Advisory Committee. MMWR
Morb Mort Wkly Rep. 2001;50:1–17.
6. Kempe A, Steiner JF, Renfrew BL, et al. How much does a regional im-
munization registry increase documented immunization rates at primary
care sites in rural Colorado? Ambul Pediatr. 2001;1:213–216.
7. Davidson A, Melinkovich P, Beatty B, et al. Immunization registry
accuracy: improvement with progressive clinical application. Am J
Prev Med. 2003;24:276 –280.
8. Centers for Disease Control and Prevention. National, state, and ur-
ban area vaccination coverage among children aged 19 –35
months—United States, 2003. MMWR Morb Mort Wkly Rep.
2004;53:658 – 661.
9. Strine TW, Barker LE, Jain RB, et al. Extraimmunization in children
through 2000. JAMA. 2002;287:588 –589.
10. Samuels RC, Appel L, Reddy SI, Tilson RS. Improving accu-
racy in a computerized immunization registry. Ambul Pediatr.
2002;2:187–192.
11. US Department of Health and Human Services. With Understanding
and Improving Health and Objectives for Improving Health:
Healthy People 2010. 2nd ed. Washington, DC: US Government
Printing Office; 2000.
12. Glazner JE, Beaty BL, Pearson KA, et al. Using an immunization
registry: effect on practice costs and time. Ambul Pediatr.
2004;4:34 – 40.
13. National Vaccine Advisory Committee. Strategies to sustain success
in childhood immunizations. JAMA. 1999;282:363–370.
14. Boyd TD, Linkins RW, Mason K, et al. Assessing immunization
registry data completeness in Bexar County, Texas. Am J Prev Med
2002;22:184 –187.
15. US Department of Health and Human Services. National Immuniza-
tion Program 2004 Annual Report: Ten Years of Achievements in
Immunization. Washington, DC: US Government Printing Office;
2000:48 –50.
16. Saarlas KN, Edwards K, Wild E, Richmond P. Developing perfor-
mance measures for immunization registries. J Public Health Man-
age Pract. 2003:9:47–57.
17. Freeman VA, DeFriese GH. The challenge and potential of childhood
immunization registries. Ann Rev Public Health. 2003;24:227–246.
18. Guadino JA, DeHart MP, Cheadle A, et al. Childhood immunization
registries: gaps between knowledge and action among family prac-
tice physicians and pediatricians in Washington state, 1998. Arch
Pediatr Adolesc Med. 2002;156:978 –985.
19. National Immunization Program. Clinic Assessment Software Appli-
cation (CASA), Modified Version 3. Atlanta, Ga: Department of
Health and Human Services; 1996.
20. Centers for Disease Control and Prevention. Notice to readers: rec-
ommended childhood immunization schedule—United States, 2000.
MMWR Morb Mort Wkly Rep. 2000;49:35–38.
21. SAS Institue Inc. SAS for Windows, Version 8.02. Cary, NC: SAS
Institute; 1999 –2001.
22. McCauley MM, Luman ET, Barker LE, Rodewald LE, Simpson
DM, Szilasyi PG. The National Immunization Survey—information
for action. Am J Prev Med. 2001;20(4 supplement):1–2.
AMBULATORY PEDIATRICS24 Kolasa et al
