A vision-based state estimation framework is proposed to localize a remotely operated vehicle (ROV) that is used to inspect a nuclear reactor pressure vessel. Using an extended Kalman filter, the framework leverages an external, pan-tilt-zoom (PTZ) camera as the primary sensing modality. In addition to the ROV and camera states, the framework also estimates a map of the reactor lower core from a prior. Both simulation and camera experiments are conducted to establish the correctness of the framework. Camera experiments (consisting of subscale and platform tests) validate the framework in terms of accuracy and robustness to environmental image degradation caused by speckling and color attenuation. Subscale mockup experiments highlight estimation consistency as compared to ground truth despite visually degraded operated conditions. Full-scale platform experiments are conducted using the actual inspection system in a dry setting. In this case, the ROV achieves a lower state uncertainty as compared to subscale mockup evaluation. For both subscale and full-scale experiments, the state uncertainty is robust to environmental image degradation effects. To further increase the efficiency and utility of the framework, we formulate and evaluate an online initialization methodology that bootstraps the framework in situ by calibrating the camera from the reactor structure. A Monte Carlo simulation shows that the initialization methodology is not sensitive to camera mounting position and orientation if the camera is placed near the center of the vessel. This framework sets the foundation for improving not only ROV-based inspection through automation, but also visual inspection with the PTZ camera through mosaics and dense reconstructions of the reactor lower core.